10 


2 8 


NATIONAL  BUREAU  OF  STANDARDS 

MICROCOPY  RESOLUTION  TEST  CHART 

l ; . 


c 


TECHNICAL  REPORT  NO. 


OTR— 62— 77— 2 
31  July  1977 


FINAL  REPORT 

RESEARCH  ON  MANNED  SYSTEM 
DESIGN  USING  OPERATOR  MEASURES 
AND  CRITERIA  (OMAC)  DATA 


E.  M.  Connelly 

D D C 

L NOV  1G  1977  ji 

jljEISlbU  u LL 

cm  f 

This  research  was  supported  by  the 
Engineering  Psychology  Programs, 

Office  of  Naval  Research 


Approved  for  Public  Release;  Distribution  Unlimited 


OMNEMII,  INC 

P.  O.  Box  745 
Vienna,  Virginia  22180 


703-938-5222 


I 


SECURITY  CLASSIFICATION  OF  THIS  PAGE  ( When  Data  Entered) 


REPORT  DOCUMENTATION  PAGE 


3 E PORT  number 


3 Arp  READ  INSTRUCTIONS 

HEFORE  COMPLETING  FORM 

2.  GOVT  ACCESSION  NO,  rS^NRECIPI  ENI.'y  C AT  ALOG  NUMBER 


OTR-62-77— 2 i ^ ( 

— \ 

4.  TITLE  (and  SutlIM.I - A 

RESEARCH  ON  MANNED  SYSTEM  DESIGN  \ 
USING  OPERATOR  MEASURES  AND  CRITERIA  1 
(OMAC)  DATA  k ^ 


L TYPE  PF  REPORT  & PERIOD  COVERED 

Final^  1 Ap  rift  14175  4^ 


REPORT  NUMBER 


AU  THORf  •) 


Edward  M. /Connelly 


f ^ a.  CONTRACT  OR  GRANT  NIJMBJfeRfa) 

UK  NOO014-75-C-p81)£>  j 


10.  PROGRAM  ELEMENT.  PROJECT,  TASK 
AREA  » WORK  UNIT  NUMBERS 


Work  Unit  NR  197-030 


12  REPQalJlAM  _ 


9.  PERFORMING  ORGANIZATION  NAME  AND  ADD/EM  10.  PROGRAM  El  EM  EN  T.  PROJ  ECT,  T ASK 

Omnemu,  Inc,  X y area  » work  unit  numbers 

P.  O.  Box  745  f U-,f  • , Work  Unit  NR  197-030 

Vienna,  VA  22180  ' _ 

11.  CONTROLLING  OFFICE  NAME  AND  ADDRESS  ~7  ~~  1 2 PfPQBT  

Office  of  Naval  Research  ( j //  31  Julg  tS77  \ / 

Department  of  the  Navy  v — ^ or  pages 

Arlington,  Virginia  22217 ££ 

14  MONITORING  AGENCY  NAME  4 AOORESS (II  dlllorool  Irom  Conlrolllnt  Olll co)  IS.  SECURITY  CLASS,  (of  l hlo  roporl) 

Engineering  Psychology  Programs,  Code  455 

_ . , , , Unclassified 

Office  of  Naval  Research  

800  North  Quincy  Street  ,s*  declassif, cation/ downgrading 


( if)  31  July  +977 

^ U.'Tlftj  STOP'S  OF  PAG 


Unclassified 


SCHEDULE 


I 16  DISTRIBUTION  STATEMENT  (of  I hit  Roporl) 


Approved  for  Public  Release;  Distribution  Unlimited 


17  DISTRIBUTION  STATEMENT  (of  the  abetract  enterad  fn  Block  20,  ft  different  from  Report) 


18  SUPPLEMENTARY  NOTES 


^ , V'\ 

\\v\ 


None 


[is  KEY#OROs7Confinu*on  taverea  aide  if  nacaeaary  and  identify  by  block  number) 


Vehicle  Control 

Adaptive  Control  Decisions 

Control-Operator  Methodology 


Control  Design 
Human  Operator  Models 


20  ABSTRACT  rCarrttnua  «n  rarer—  aidm  ft  nmcmmamry  mad  Identify  by  block  number) 

In  manned  systems,  performance  can  change  significantly  with  changes 
in  display  design.  With  today's  computer  and  display  technology,  it  is  possible 
to  provide  virtually  any  display  function  desired  including  automating  many  of 
the  information  processing  tasks  previously  performed  by  the  human  operator. 
However,  the  relationship  between  display  design  and  total  system  (people 
and  machine)  performance  must  be  known  in  order  to  systematically  select 
display  matures. he  object  of  this  research  program  was  to  invest!  nam 


DO  M73  EDITION  OF  I NOV  65  IS  OBSOLETE 


y 1 70  t> 


Unclassified 

SECURITY  CLASSIFICATION  OF  THIS  PAGE  Dal*  Entorod) 


Unclassified 


SECURITY  CLASSIFICATION  OF  THIS  PAGE(T»»l«n  Dmtm  Enfr.J)  


system  performance  models  for  ship  control  as  an  aid  to  ship  display  and 
control  design. 

A human  operator  model , which  represents  the  total  system  response  by 
identifying  the  criteria  optimized  by  that  response,  was  developed  to  represent 
the  ship  control  performance  of  the  Officer  of  the  Deck  (OOD).  In  addition,  a 
sensitive  contact  (ship)  avoidance  measure  was  developed  which  detects  con- 
ditions leading  to  ship  collisions  and  near  collisions. ^ The  OMAC  and  perfor- 
mance measure  were  used  to  demonstrate  that  significantly  improved  per- 
formance can  be  obtained  with  a new  display  design  that  automates  information 
processing  previously  required  of  the  OOD. 


->40MAC  models  representing  performance  obtained  with  each  display  design 
reveal  that  performance  differences  are  explained  by  differences  in  a con- 
straint self-imposed  by  the  operator  to  select  only  a portion  of  the  display 
information  in  order  to  control  the  ship.  Constraint  differences  are  equivalent 
to  differences  in  the  amount  of  information  processed  by  the  OOD  with  each 
display  design.*  Further,  the  hypothesis  that  OOD  participants  using  different 
displays  attempt  to  perform  according  to  invariant  performance  criteria  was 
confirmed  for  superior  performances . The  hypothesis  was  not  confirmed  for 
less  than  superior  performances. 


c Sfdion 
B.i;  Secton  D 


i"  <'J$$ 


Line 


SECURITY  CLASSIFICATION  OF  THIS  P AGEfWi«n  D»fa  Enfrmd) 


TABLE  OF  CONTENTS 


SECTION  PAGE 

1.0  INTRODUCTION  1-1 

2.0  MAJOR  RESULTS  OF  THE  RESEARCH  2-1 

2.1  Contact  Avoidance  Performance  Measurement  2-1 

2.2  OMAC  Modeling  Concept  2-2 

2.3  OMAC  Representation  of  Operator  Performances  2-4 

2.4  OMAC  Representation  of  Performance  With  2-5 

OLD  and  PACS  Displays 

2.5  Representation  of  Human  Performance  by  the  2-7 

OMAC  Optimized 


Reports  Distributed 
References 
Distribution  List 


ABSTRACT 


In  manned  systems,  performance  can  change  significantly 
with  changes  in  display  design.  With  today's  computer  and  display 
technology,  it  is  possible  to  provide  virtually  any  display  function 
desired  including  automating  many  of  the  information  processing  tasks 
previously  performed  by  the  human  operator.  However,  the  relation- 
ship between  display  design  and  total  system  (people  and  machine) 
performance  must  be  known  in  order  to  systematically  select  display 
features.  The  object  of  this  research  program  was  to  investigate 
system  performance  models  for  ship  control  as  an  aid  to  ship  display 
and  control  design. 

A human  operator  model,  which  represents  the  total 
system  response  by  identifying  the  criteria  optimized  by  that  response, 
was  developed  to  represent  the  ship  control  performance  of  the  Officer 
of  the  Deck  (OOD).  In  addition,  a sensitive  contact  (ship)  avoidance 
measure  was  developed  which  detects  conditions  leading  to  ship 
collisions  and  near  collisions.  The  OMAC  and  performance  measure 
were  used  to  demonstrate  that  significantly  improved  performance 
can  be  obtained  with  a new  display  design  that  automates  information 
processing  previously  required  of  the  OOD. 

OMAC  models  representing  performance  obtained  with 
each  display  design  reveal  that  performance  differences  are  explained 
by  differences  in  a constraint  self-imposed  by  the  operator  to  select 
only  a portion  of  the  display  information  in  order  to  control  the  ship. 
Constraint  differences  are  equivalent  to  differences  in  the  amount  of 
information  processed  by  the  OOD  with  each  display  design.  Further, 
the  hypothesis  that  OOD  participants  using  different  displays  attempt  to 
perform  according  to  invariant  performance  criteria  was  confirmed 
for  superior  performances . The  hypothesis  was  not  confirmed  for  less 
than  superior  performances . 
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INTRODUCTION 


This  report  is  the  final  report  on  contract  number 
N0001 4-75— C-081 0 between  Omnemii,  Inc.  and  Engineering 
Psychology  Programs,  Office  of  Naval  Research.  The  contract 
was  initiated  on  1 April  1975  and  completed  on  30  May  1977. 

The  objectives  of  the  research  were  to: 

1 . Develop  a ship  display/control  design  tool 
which  would  permit  a designer  to  select 
alternative  display  features  based  on  their 
effect  on  system  performance. 

2.  Develop  a method  of  representing  operator 
control  actions  and  resulting  ship  responses 
by  the  criteria  optimized  by  those  ship 
responses.  This  type  of  model  is  called 
"operator  measures  and  criteria"  (OMAC) 
model . 

3.  Devise  a method  of  using  the  OMAC  model  to 
predict  operator  control  actions  and  resultant 
ship  responses  in  a variety  of  problem  situations. 

4.  Develop  a sensitive  measure  of  contact  (ship) 
avoidance  performance. 
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This  research  program  used  data  which  had  been 


previously  collected  during  a series  of  experiments  in  which 
participants  acting  as  Officer  of  the  Deck  (OOD)  controlled  a 
simulated  ship  in  a simulated  environment.  The  task  was  to  direct 
a ship  transit  from  the  initial  point  to  the  terminal  point  within 
a pre-specified  time  interval  while  avoiding  simulated  contacts 
along  the  way.  The  experiment,  using  equipment  known  as  the 
Surface  Ship  Bridge  Console  System  (Beary;  Gawitt),  was  run  by 
personnel  of  NSRDC,  Annapolis,  Maryland,  for  purposes  other  than 
this  research  program.  The  data  from  that  experiment  were  used 
in  the  research  reported  here. 

Reports  describing  the  research  are  listed  in  the  section 
on  reports  distributed.  Results  of  the  research  are  summarized  in 
the  following  paragraphs. 
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2.0 


MAJOR  RESULTS  OF  THE  RESEARCH 


2 . 1 Contact  Avoidance  Performance  Measurement 

Performance  of  manned  systems  is  typically  measured 
with  summary  measures,  i.e.,  measures  that  summarize  performance 
over  the  total  simulated  or  actual  mission.  Examples  of  such  measure 
components  applied  to  ship  control  are: 

1 . T ransit  time; 

2.  Average  deviation  from  a reference  course; 

3.  Number  of  collisions  or  near  collisions; 

4.  Fuel  efficiency;  and 

5.  Number  of  course  and  speed  changes. 

These  components  are  usually  weighted  and  summed  to  form  the 
composit  summary  measure.  However,  such  summary  measures 
did  not  reveal  the  performance  differences  that  existed  with  different 
displays  tested  in  the  ship  control  experiments  referred  to  previously. 
Fortunately,  a sensitive  contact  avoidance  measure  was  developed  as 
part  of  the  research  on  this  program  to  identify  OMAC's  that  represent 
operator  performances . With  this  measure,  the  performance  differences 
with  different  displays  were  revealed. 

Of  importance  here  is  not  only  discovery  of  improved 
performance  with  the  advanced  displays,  but  also  the  development  of 
the  sensitive  contact  avoidance  performance  measure.  The  success  of 
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the  measure  supports  the  measurement  principle:  measures  that 
detect  responses  leading  to  a critical  condition  (a  ship  collision) 
are  more  sensitive  than  measures  that  detect  only  the  critical  condition. 
2.2  OMAC  Modeling  Concept 

Performance  prediction  models  have  long  been  sought  to 
aid  in  system  design  and  to  better  understand  how  display  designs 
affect  performance.  Present  modeling  methods  (Connelly;  Kleinman; 
McRuer;  Preyss;  Sheridan)  which  typically  represent  the  human 
operator's  moment-to-moment  control  response,  have  not  provided 
the  necessary  predictive  models.  There  are  several  difficult  pro- 
blems to  overcome.  One  problem  is  that  performance  models  should 
be  applicable  to  operational  or  near  operational  settings  and  not 
limited  to  laboratory  problems.  Operational  settings  frequently 
require  non-linear  operator  response  functions  - thus  prohibiting  the 
use  of  linear  models.  Another  problem  is  that  prediction  of  human 
response  or  human  controlled  system  response  on  a moment-by-moment 
basis  does  not  permit  prediction  of  present  response  based  on  present 
conditions  and  the  anticipated  or  planned  activity  by  the  operator.  The 
modeling  method  used  in  the  analysis  reported  here  employs  a construct 
based  on  optimal  control  theory  to  provide  performance  prediction. 
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Optimal  control  theory  provides  a means  for  determining 
the  system  response  which,  within  the  constraints  imposed,  is  best 
| according  to  the  specified  criteria.  Given  criteria  and  constraints, 

optimal  control  theory  permits  evaluation  of  all  possible  responses 
that  satisfy  the  constraints,  and  in  doing  so  identified  the  response 

I 

that  is  best  according  to  the  criteria.  Its  property  of  interest  here 
is  that  it  relates  system  responses  to  criteria  and  vice  versa.  With 
this  capability,  it  is  possible  to  model  a response  by  identifying  the 
criteria  optimized.  This  modeling  process  is  the  inverse  of  the 
process  used  by  the  designer  of  an  automatic  control  system  wherein 
criteria  are  selected  first  followed  by  determination  of  system  responses 
that  optimize  those  criteria.  The  inverse  process  used  in  this  study 
starts  with  observed  responses  and  seeks  to  identifv  the  criteria 


optimized  by  these  responses. 

For  manned  systems,  the  OMAC  criteria  found  to  be 
optimized  by  observed  responses  should  be  distinguished  from  task 
criteria  typically  established  by  the  system  designer  to  specify 
desired  system  performance.  While  the  OMAC  criteria  and  the  task 
criteria  may  be  identical  or  similar,  they  are  not  necessarily  the 
same;  differences  would  reflect  a lack  of  mutual  understanding  of  task 
performance  requirements . Also  regarding  manned  systems,  constraints 
can  be  imposed  by  the  equipment  (e.g.,  limited  rudder  deflection)  or 
imposed  by  the  operator  (e.g.  , use  of  only  a portion  of  the  display  information). 
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The  control  task  considered  here  is  a ship  control 


problem  where  the  OOD  is  to  direct  the  ship  from  a starting  location 
to  an  objective  location  and  arrive  at  the  objective  in  90  minutes. 

The  ship  is  to  be  controlled  so  as  to  avoid  all  contacts  (other  ships) 
by  at  least  3.70  kilometers  (2  nautical  miles).  A task  performance 
measure  for  this  problem  was  developed  using  measures  of  transit 
time  and  contact  avoidance.  This  task  measure  penalizes  for  excessive 
transit  time  and  for  passing  contacts  closer  than  3.70  kilometers 
(2  nautical  miles).  The  ship  control  involves  equipment  constraint 
such  as  an  upper  limit  to  speed  and  a lower  limit  to  turning  radius. 

Using  the  OMAC  criteria  analysis  described,  ship  transiting  responses 
which  included  contact  avoidance  maneuvering  were  modeled  by  the 
criteria  optimized. 

2.3  OMAC  Representation  of  Operator  Performances 

OMAC's  were  identified  for  each  operator  performance.  The 
ability  of  the  OMAC's  to  represent  individual  performances  was  shown 
to  be  a function  of  the  level  of  operator  performance,  i.e.,  superior 
performers  were  represented  more  accurately  than  were  less  skillful 
performers.  In  spite  of  difference  in  accuracy  of  representation,  the 
representation  is  sufficiently  accurate  to  permit  prediction  of  operator 
performance,  i.e.,  prediction  of  operator  performance  using  OMAC's 
in  a statistically  significant  manner. 
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A major  result  of  the  research  was  that  for  the  ship 


control  problem  considered,  OMAC  must  include  both  the  criteria 

optimized  and  a purview  constraint.  Without  both  criteria  and 

constraint,  accurate  modeling  was  not  possible.  The  purview 

constraint  found  useful  in  the  OMAC  model  represents  a radius 

from  own  ship  within  which  all  contacts  are  considered.  Contacts 

outside  that  radius  are  not  considered  for  ship  control . Other 

constraints  such  as  considering  a limited  number  of  contacts  may 

be  useful  but  some  type  of  purview  function  is  necessary. 

2.4  OMAC  Representation  of  Performance  With  OLD 

and  PACS  Displays 

OMAC's  representing  operator  performance  with  two 
different  display  types  (OLD  and  PACS^)  for  two  population  groups 
[all  participants  and  superior  participants)  were  identified.  From 
the  results,  several  conclusions  were  possible. 

1 . A greater  proportion  of  the  performers  were 
rated  superior  with  the  PACS  display  than  with  the  OLD  display. 

This  suggests  that  with  the  PACS  display  there  would  be  more  superior 
performers  in  the  general  OOD  population. 

2.  Superior  performers  with  PACS  and  OLD  displays 
are  represented  by  the  same  OMAC  criteria  (the  relative  weighting 

'old  display  is  a conventional  radar  PPI  display,  PACS  (Possible  Area  of 
Collision)  is  an  advanced  display  that  provides  among  other  features  the  locus 
of  collisions  with  all  contacts  for  all  possible  own  ship  courses. 
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between  contact  avoidance  and  transit  time).  The  average  performance 
level  with  the  two  displays  is  different  but  the  criteria  optimized  are 
the  same.  This  suggests  that  consistent  "target"  criteria  (the  criteria 
sought  by  the  operator)  were  the  goal  of  superior  performers  with  both 
displays;  but,  the  operators  were  better  able  to  reach  that  performance 
goal  with  the  PACS  display.  This  result  suggests  that  the  "target" 
criteria  might  be  used  in  training  where  operators  could  be  rated  not 
only  on  performance  but  also  on  the  criteria  they  apparently  optimize. 

3.  Performance  differences  with  different  display  types 
obtained  from  the  superior  performances  are  explained  by  a difference 

in  purview  (the  range  from  own  ship  within  which  the  operator  processes 
contact  data).  The  logic  is:  OMAC  predicts  operator  performance; 

OMAC  has  two  parts:  criteria  and  purview;  but,  the  criteria  are 
constant  while  purview  changes  with  changes  in  display.  Indeed,  the 
OMAC  performance  data  show  clearly  the  effect  of  purview  on  per- 
formance. 

4.  The  purview  rating  for  the  OLD  display  was  found 
to  be  17.77  kilometers  (9.6  nautical  miles)  while  the  PACS  display 
was  found  to  be  22.22  kilometers  (12  nautical  miles).  Purview  rating 
may  be  a function  of  contact  density  and  thus  the  ratings  given  should 

not  be  considered  absolute;  but,  only  apply  for  the  experimental  control  task. 
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5.  The  average  number  of  contacts  within  purview  in 
the  experimental  problem  was  a linear  function  of  purview  area.  Thus, 
OOD  ship  handling  performance  may  be  a function  of  the  number  of 
contacts  that  the  OOD  can  process  with  a given  display  design.  If  a 
display  feature  automates  one  or  more  information  processing  tasks, 
the  OOD  may  be  able  to  process  more  contacts,  thus  expanding  his 
purview.  According  to  the  results  discussed  above  (3),  this  would 
result  in  improved  performance.  This  logic  suggests  that  an 
analysis  of  the  information  processing  required  of  the  OOD  per 
contact  may  permit  direct  prediction  of  OOD  ship  handling  performance. 

2 . b Representation  of  Human  Performance  by  the  QMAC  Optimized 

Based  on  the  overall  results  obtained  with  the  OMAC,  it 
is  concluded  that  the  criteria  modeling  approach  offers  a practical  way 
to  model  human  performance  in  an  operational  or  near  operational 
problem  setting.  The  generality  of  the  method  to  other  manned  system 
problems  should  be  investigated. 

With  this  modeling  method,  the  operator's  output  at  each 
instant  of  time  is  represented  by  taking  into  account  both  the  instantaneous 
problem  state  and  planned  operator  outputs  as  a result  of  projections  of 
controlled  device  responses.  Projections  of  controlled  device  responses 
are  the  expected  future  responses  of  the  device(s)  or  expected  behavior 
of  disturbing  factors  such  as  the  contacts  of  the  ship  control  problem. 
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Representative  OMAC's  reveal  the  target  criteria  of  the 
human  controller  and  permit  evaluation  of  the  ability  to  perform 
according  to  those  criteria.  OMAC's  also  reveal  the  constraints 
that  may  be  self-imposed  or  are  inherent,  but  which  limit  the 
operator's  performance  level. 

OMAC's  can  be  developed  for  individuals  working  with 
complex  and  non-linear  tasks.  The  complexity  of  the  task  or  the 
device  being  controlled  does  not  limit  the  identification  of  the 
apparent  OMAC . 
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